Prior-exposure-test Experiment

Catch trial performance

## [1] "Excluded 1 participants based on catch-trial performance."

Exclusion of random guesses

We further exclude participants who seem to provide random ratings independent of the scene that they are seeing. We quantify this by computing the mean rating for each utterance across all trials for each participant and computing the correlation between a participant’s actual ratings and their mean rating. A high correlation is unexpected and indicates that a participant chose ratings at random. We therefore also exclude the data from participants for whom this correlation is larger than 0.75.

## `summarise()` has grouped output by 'modal'. You can override using the
## `.groups` argument.
## `summarise()` has grouped output by 'modal', 'percentage_blue'. You can
## override using the `.groups` argument.

## [1] "Excluded 0 participants based on random responses."

Aggregated results

## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'percentage_blue', 'modal'. You can
## override using the `.groups` argument.
## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.

Comparison across conditions

## `summarise()` has grouped output by 'workerid', 'percentage_blue', 'modal'. You
## can override using the `.groups` argument.
## `summarise()` has grouped output by 'percentage_blue', 'modal'. You can
## override using the `.groups` argument.

Individual responses

AUC computation

We use the AUC function with the splines method to directly compute the AUC.

t-test and regression model with control variables:

## 
##  Two Sample t-test
## 
## data:  aucs.cautious$auc_diff and aucs.confident$auc_diff
## t = 2.9517, df = 122, p-value = 0.003791
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##   3.969716 20.138118
## sample estimates:
## mean of x mean of y 
##  20.20750   8.15358

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: 
## auc_diff ~ cond + test_order + first_speaker_type + confident_speaker +  
##     first_speaker_type * cond + (1 | workerid)
##    Data: auc_d
## 
## REML criterion at convergence: 1086.2
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.1828 -0.4581  0.0770  0.6374  2.0920 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  workerid (Intercept)  89.25    9.447  
##  Residual             376.89   19.414  
## Number of obs: 124, groups:  workerid, 62
## 
## Fixed effects:
##                           Estimate Std. Error     df t value Pr(>|t|)    
## (Intercept)                 14.152      2.118 58.000   6.684 9.95e-09 ***
## cond1                        6.027      1.743 60.000   3.457  0.00101 ** 
## test_order1                  0.878      2.122 58.000   0.414  0.68056    
## first_speaker_type1         -7.666      2.118 58.000  -3.620  0.00062 ***
## confident_speaker1           2.544      2.122 58.000   1.199  0.23544    
## cond1:first_speaker_type1    1.211      1.743 60.000   0.695  0.48990    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) cond1  tst_r1 frs__1 cnfd_1
## cond1        0.000                            
## test_order1 -0.032  0.000                     
## frst_spkr_1  0.000  0.000  0.002              
## cnfdnt_spk1  0.002  0.000 -0.065 -0.032       
## cnd1:frs__1  0.000  0.000  0.000  0.000  0.000

Clustering analyses

library(mclust)

## Package 'mclust' version 5.4.10
## Type 'citation("mclust")' for citing this R package in publications.

## 
## Attaching package: 'mclust'

## The following object is masked from 'package:DescTools':
## 
##     BrierScore

## The following object is masked from 'package:bootstrap':
## 
##     diabetes

aucs_diff = merge(aucs.cautious, aucs.confident, by=c("workerid"))
aucs_diff$diff_of_diffs = aucs_diff$auc_diff.x - aucs_diff$auc_diff.y

aucs_diff %>% ggplot(aes(x=diff_of_diffs)) + geom_density() + geom_jitter(aes(y=0), width=0, height=0.001)  + ggtitle("Raw data + estimated density")

Gaussian mixture models of diffeences of AUC differences

1 Cluster

fit1 = Mclust(aucs_diff$diff_of_diffs, G=1)
print(summary(fit1, parameters=2))

## ---------------------------------------------------- 
## Gaussian finite mixture model fitted by EM algorithm 
## ---------------------------------------------------- 
## 
## Mclust X (univariate normal) model with 1 component: 
## 
##  log-likelihood  n df      BIC      ICL
##       -292.5844 62  2 -593.423 -593.423
## 
## Clustering table:
##  1 
## 62 
## 
## Mixing probabilities:
## 1 
## 1 
## 
## Means:
## [1] 12.05392
## 
## Variances:
## [1] 735.3371

2 Clusters

fit2 = Mclust(aucs_diff$diff_of_diffs, G=2)
print(summary(fit2, parameters=T))

## ---------------------------------------------------- 
## Gaussian finite mixture model fitted by EM algorithm 
## ---------------------------------------------------- 
## 
## Mclust E (univariate, equal variance) model with 2 components: 
## 
##  log-likelihood  n df       BIC       ICL
##       -288.1198 62  4 -592.7482 -598.9994
## 
## Clustering table:
##  1  2 
## 55  7 
## 
## Mixing probabilities:
##        1        2 
## 0.864934 0.135066 
## 
## Means:
##         1         2 
##  4.335758 61.479346 
## 
## Variances:
##        1        2 
## 353.8638 353.8638

3 Clusters

fit3 = Mclust(aucs_diff$diff_of_diffs, G=3)
print(summary(fit3, parameters=T))

## ---------------------------------------------------- 
## Gaussian finite mixture model fitted by EM algorithm 
## ---------------------------------------------------- 
## 
## Mclust E (univariate, equal variance) model with 3 components: 
## 
##  log-likelihood  n df       BIC       ICL
##       -288.1317 62  6 -601.0263 -656.7775
## 
## Clustering table:
##  1  2  3 
##  9 46  7 
## 
## Mixing probabilities:
##         1         2         3 
## 0.3373576 0.5314408 0.1312016 
## 
## Means:
##         1         2         3 
## -1.081510  7.998816 62.254385 
## 
## Variances:
##        1        2        3 
## 337.7512 337.7512 337.7512

According to the Bayesian information criterion, a model with two clusters describes the data best.

Fitted model:

aucs_diff %>% 
  ggplot(aes(x=diff_of_diffs)) + 
    geom_jitter(aes(y=0, color=first_speaker_type.x), width=0, height=0.001)  +
    ggtitle("Raw data + Components of gaussian mixture") + 
    stat_function(fun = dnorm, args = list(mean = fit2$parameters$mean[1], sd = sqrt(fit2$parameters$variance$sigmasq[1]))) + 
    stat_function(fun = dnorm, args = list(mean = fit2$parameters$mean[2], sd = sqrt(fit2$parameters$variance$sigmasq[2])))

## Warning: Removed 101 row(s) containing missing values (geom_path).

Compute likelihoods based on the adaptation model

## Generalized linear mixed model fit by maximum likelihood (Laplace
##   Approximation) [glmerMod]
##  Family: binomial  ( logit )
## Formula: most_likely_model ~ condition + test_order + first_speaker_type +  
##     first_speaker_type * condition + (1 | workerid)
##    Data: d.post_test
## 
##      AIC      BIC   logLik deviance df.resid 
##    154.1    171.0    -71.0    142.1      118 
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -1.6887 -0.5686 -0.2379  0.5566  2.2970 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  workerid (Intercept) 1.849    1.36    
## Number of obs: 124, groups:  workerid, 62
## 
## Fixed effects:
##                                              Estimate Std. Error z value
## (Intercept)                                   -0.4481     0.3106  -1.443
## conditioncautious                             -0.8698     0.2906  -2.993
## test_orderparallel                            -0.4682     0.3081  -1.520
## first_speaker_typecautious                     0.8680     0.3414   2.542
## conditioncautious:first_speaker_typecautious  -0.2066     0.2423  -0.852
##                                              Pr(>|z|)   
## (Intercept)                                   0.14906   
## conditioncautious                             0.00276 **
## test_orderparallel                            0.12856   
## first_speaker_typecautious                    0.01102 * 
## conditioncautious:first_speaker_typecautious  0.39396   
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) cndtnc tst_rd frst__
## conditincts  0.209                     
## tst_rdrprll  0.106  0.220              
## frst_spkr_t -0.193 -0.363 -0.198       
## cndtncts:__ -0.039  0.080  0.055 -0.062

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: likelihood_ratio ~ condition + test_order + first_speaker_type +  
##     first_speaker_type * condition + (1 | workerid)
##    Data: d.post_test
## 
## REML criterion at convergence: 1596.8
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.2024 -0.6789 -0.0483  0.5532  2.6459 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  workerid (Intercept)  2007     44.8   
##  Residual             30231    173.9   
## Number of obs: 124, groups:  workerid, 62
## 
## Fixed effects:
##                                              Estimate Std. Error     df t value
## (Intercept)                                    -14.48      16.63  59.00  -0.871
## conditioncautious                              -56.74      15.61  60.00  -3.634
## test_orderparallel                             -25.21      16.63  59.00  -1.516
## first_speaker_typecautious                      56.68      16.62  59.00   3.410
## conditioncautious:first_speaker_typecautious   -13.92      15.61  60.00  -0.891
##                                              Pr(>|t|)    
## (Intercept)                                   0.38744    
## conditioncautious                             0.00058 ***
## test_orderparallel                            0.13477    
## first_speaker_typecautious                    0.00118 ** 
## conditioncautious:first_speaker_typecautious  0.37634    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) cndtnc tst_rd frst__
## conditincts  0.000                     
## tst_rdrprll -0.032  0.000              
## frst_spkr_t  0.000  0.000  0.000       
## cndtncts:__  0.000  0.000  0.000  0.000

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: likelihood_ratio ~ condition + test_order + first_speaker_type +  
##     prior_likelihood_ratio + first_speaker_type * condition +  
##     (1 | workerid)
##    Data: d.post_test
## 
## REML criterion at convergence: 1587.1
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.39818 -0.66926 -0.04121  0.51524  3.06625 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  workerid (Intercept)     0      0.0   
##  Residual             29267    171.1   
## Number of obs: 124, groups:  workerid, 62
## 
## Fixed effects:
##                                              Estimate Std. Error       df
## (Intercept)                                   16.4539    17.6008 118.0000
## conditioncautious                            -56.7419    15.3630 118.0000
## test_orderparallel                           -20.5737    15.4247 118.0000
## first_speaker_typecautious                    49.0672    15.5072 118.0000
## prior_likelihood_ratio                         0.4202     0.1165 118.0000
## conditioncautious:first_speaker_typecautious -13.9165    15.3630 118.0000
##                                              t value Pr(>|t|)    
## (Intercept)                                    0.935 0.351779    
## conditioncautious                             -3.693 0.000337 ***
## test_orderparallel                            -1.334 0.184835    
## first_speaker_typecautious                     3.164 0.001979 ** 
## prior_likelihood_ratio                         3.607 0.000455 ***
## conditioncautious:first_speaker_typecautious  -0.906 0.366866    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr) cndtnc tst_rd frst__ prr_l_
## conditincts  0.000                            
## tst_rdrprll  0.013  0.000                     
## frst_spkr_t -0.066  0.000 -0.011              
## prr_lklhd_r  0.487  0.000  0.083 -0.136       
## cndtncts:__  0.000  0.000  0.000  0.000  0.000
## optimizer (nloptwrap) convergence code: 0 (OK)
## boundary (singular) fit: see help('isSingular')

## Data: d.post_test
## Models:
## model1: likelihood_ratio ~ condition + test_order + first_speaker_type + first_speaker_type * condition + (1 | workerid)
## model2: likelihood_ratio ~ condition + test_order + first_speaker_type + prior_likelihood_ratio + first_speaker_type * condition + (1 | workerid)
##        npar    AIC    BIC  logLik deviance  Chisq Df Pr(>Chisq)    
## model1    7 1647.8 1667.5 -816.89   1633.8                         
## model2    8 1637.0 1659.5 -810.49   1621.0 12.792  1  0.0003481 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

List of adapters:

workerid	first_speaker_type	test_order	noticed_manipulation	cautious_count	confident_count	aligned_count	first_adaptation_speaker_count
1352	cautious	reverse	1	1	1	2	1
1353	confident	parallel	0	1	1	2	1
1354	cautious	parallel	0	1	1	2	1
1355	confident	reverse	0	1	1	2	1
1359	confident	reverse	0	1	1	2	1
1365	cautious	reverse	0	1	1	2	1
1368	confident	parallel	1	1	1	2	1
1370	confident	parallel	1	1	1	2	1
1373	cautious	reverse	1	1	1	2	1
1375	confident	reverse	1	1	1	2	1
1385	cautious	reverse	1	1	1	2	1
1387	cautious	reverse	1	1	1	2	1
1391	cautious	reverse	1	1	1	2	1
1395	cautious	parallel	1	1	1	2	1
1397	cautious	reverse	1	1	1	2	1
1407	confident	reverse	0	1	1	2	1
1417	cautious	reverse	1	1	1	2	1
1423	cautious	parallel	1	1	1	2	1
1428	cautious	parallel	1	1	1	2	1
1432	confident	reverse	0	1	1	2	1

List of reverse adapters:

workerid	first_speaker_type	test_order	noticed_manipulation	cautious_count	confident_count	aligned_count	first_adaptation_speaker_count
1404	confident	parallel	1	1	1	0	1
1406	cautious	parallel	0	1	1	0	1
1419	confident	reverse	0	1	1	0	1